{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description：\n",
    "这个笔记本要做一个GBDT+LR的demon， 基于kaggle上的一个比赛数据集, 下载链接：[http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/) 数据集介绍：\n",
    "这是criteo-Display Advertising Challenge比赛的部分数据集， 里面有train.csv和test.csv两个文件：\n",
    "* train.csv： 训练集由Criteo 7天内的部分流量组成。每一行对应一个由Criteo提供的显示广告。为了减少数据集的大小，正(点击)和负(未点击)的例子都以不同的比例进行了抽样。示例是按时间顺序排列的\n",
    "* test.csv: 测试集的计算方法与训练集相同，只是针对训练期之后一天的事件\n",
    "\n",
    "字段说明：\n",
    "* Label： 目标变量， 0表示未点击， 1表示点击\n",
    "* l1-l13: 13列的数值特征， 大部分是计数特征\n",
    "* C1-C26: 26列分类特征， 为了达到匿名的目的， 这些特征的值离散成了32位的数据表示\n",
    "\n",
    "这个比赛的任务就是：开发预测广告点击率(CTR)的模型。给定一个用户和他正在访问的页面，预测他点击给定广告的概率是多少？比赛的地址链接：[https://www.kaggle.com/c/criteo-display-ad-challenge/overview](https://www.kaggle.com/c/criteo-display-ad-challenge/overview)\n",
    "<br><br>\n",
    "下面基于GBDT+LR模型完后这个任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:02:40.455261Z",
     "start_time": "2020-09-10T12:02:39.318459Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"导入包\"\"\"\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "import lightgbm as lgb\n",
    "from sklearn.preprocessing import MinMaxScaler, OneHotEncoder, LabelEncoder\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "import gc\n",
    "from scipy import sparse\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据导入与简单处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:02:41.574600Z",
     "start_time": "2020-09-10T12:02:41.533663Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"数据读取与预处理\"\"\"\n",
    "\n",
    "# 数据读取\n",
    "path = 'data/'\n",
    "df_train = pd.read_csv(path + 'train.csv')\n",
    "df_test = pd.read_csv(path + 'test.csv')\n",
    "\n",
    "# 简单的数据预处理\n",
    "# 去掉id列， 把测试集和训练集合并， 填充缺失值\n",
    "df_train.drop(['Id'], axis=1, inplace=True)\n",
    "df_test.drop(['Id'], axis=1, inplace=True)\n",
    "\n",
    "df_test['Label'] = -1\n",
    "\n",
    "data = pd.concat([df_train, df_test])\n",
    "data.fillna(-1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:02:42.184548Z",
     "start_time": "2020-09-10T12:02:42.180557Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"下面把特征列分开处理\"\"\"\n",
    "continuous_fea = ['I'+str(i+1) for i in range(13)]\n",
    "category_fea = ['C'+str(i+1) for i in range(26)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建模\n",
    "下面训练三个模型对数据进行预测， 分别是LR模型， GBDT模型和两者的组合模型， 然后分别观察它们的预测效果， 对于不同的模型， 特征会有不同的处理方式如下：\n",
    "1. 逻辑回归模型： 连续特征要归一化处理， 离散特征需要one-hot处理\n",
    "2. GBDT模型： 树模型连续特征不需要归一化处理， 但是离散特征需要one-hot处理\n",
    "3. LR+GBDT模型： 由于LR使用的特征是GBDT的输出， 原数据依然是GBDT进行处理交叉， 所以只需要离散特征one-hot处理\n",
    "\n",
    "下面就通过函数的方式建立三个模型， 并进行训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 逻辑回归建模"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:02:44.358065Z",
     "start_time": "2020-09-10T12:02:44.350087Z"
    }
   },
   "outputs": [],
   "source": [
    "def lr_model(data, category_fea, continuous_fea):\n",
    "    # 连续特征归一化\n",
    "    scaler = MinMaxScaler()\n",
    "    for col in continuous_fea:\n",
    "        data[col] = scaler.fit_transform(data[col].values.reshape(-1, 1))\n",
    "    \n",
    "    # 离散特征one-hot编码\n",
    "    for col in category_fea:\n",
    "        onehot_feats = pd.get_dummies(data[col], prefix=col)\n",
    "        data.drop([col], axis=1, inplace=True)\n",
    "        data = pd.concat([data, onehot_feats], axis=1)\n",
    "    \n",
    "    # 把训练集和测试集分开\n",
    "    train = data[data['Label'] != -1]\n",
    "    target = train.pop('Label')\n",
    "    test = data[data['Label'] == -1]\n",
    "    test.drop(['Label'], axis=1, inplace=True)\n",
    "    \n",
    "    # 划分数据集\n",
    "    x_train, x_val, y_train, y_val = train_test_split(train, target, test_size=0.2, random_state=2020)\n",
    "    \n",
    "    # 建立模型\n",
    "    lr = LogisticRegression()\n",
    "    lr.fit(x_train, y_train)\n",
    "    tr_logloss = log_loss(y_train, lr.predict_proba(x_train)[:, 1])   # −(ylog(p)+(1−y)log(1−p)) log_loss\n",
    "    val_logloss = log_loss(y_val, lr.predict_proba(x_val)[:, 1])\n",
    "    print('tr_logloss: ', tr_logloss)\n",
    "    print('val_logloss: ', val_logloss)\n",
    "    \n",
    "    # 模型预测\n",
    "    y_pred = lr.predict_proba(test)[:, 1]  # predict_proba 返回n行k列的矩阵，第i行第j列上的数值是模型预测第i个预测样本为某个标签的概率, 这里的1表示点击的概率\n",
    "    print('predict: ', y_pred[:10]) # 这里看前10个， 预测为点击的概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:02:48.767635Z",
     "start_time": "2020-09-10T12:02:45.774402Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tr_logloss:  0.12423395164788172\n",
      "val_logloss:  0.4440724569877669\n",
      "predict:  [0.44783059 0.80628705 0.1756691  0.02070154 0.13984202 0.46490042\n",
      " 0.43386417 0.07089967 0.07121148 0.27896238]\n"
     ]
    }
   ],
   "source": [
    "# 训练和预测\n",
    "lr_model(data.copy(), category_fea, continuous_fea)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GBDT 建模"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:03:01.725802Z",
     "start_time": "2020-09-10T12:03:01.717823Z"
    }
   },
   "outputs": [],
   "source": [
    "def gbdt_model(data, category_fea, continuous_fea):\n",
    "    \n",
    "    # 离散特征one-hot编码\n",
    "    for col in category_fea:\n",
    "        onehot_feats = pd.get_dummies(data[col], prefix=col)\n",
    "        data.drop([col], axis=1, inplace=True)\n",
    "        data = pd.concat([data, onehot_feats], axis=1)\n",
    "    \n",
    "    # 训练集和测试集分开\n",
    "    train = data[data['Label'] != -1]\n",
    "    target = train.pop('Label')\n",
    "    test = data[data['Label'] == -1]\n",
    "    test.drop(['Label'], axis=1, inplace=True)\n",
    "    \n",
    "    # 划分数据集\n",
    "    x_train, x_val, y_train, y_val = train_test_split(train, target, test_size=0.2, random_state=2020)\n",
    "    \n",
    "    # 建模\n",
    "    gbm = lgb.LGBMClassifier(boosting_type='gbdt',  # 这里用gbdt\n",
    "                             objective='binary', \n",
    "                             subsample=0.8,\n",
    "                             min_child_weight=0.5, \n",
    "                             colsample_bytree=0.7,\n",
    "                             num_leaves=100,\n",
    "                             max_depth=12,\n",
    "                             learning_rate=0.01,\n",
    "                             n_estimators=10000\n",
    "                            )\n",
    "    gbm.fit(x_train, y_train, \n",
    "            eval_set=[(x_train, y_train), (x_val, y_val)], \n",
    "            eval_names=['train', 'val'],\n",
    "            eval_metric='binary_logloss',\n",
    "            early_stopping_rounds=100,\n",
    "           )\n",
    "    \n",
    "    tr_logloss = log_loss(y_train, gbm.predict_proba(x_train)[:, 1])   # −(ylog(p)+(1−y)log(1−p)) log_loss\n",
    "    val_logloss = log_loss(y_val, gbm.predict_proba(x_val)[:, 1])\n",
    "    print('tr_logloss: ', tr_logloss)\n",
    "    print('val_logloss: ', val_logloss)\n",
    "    \n",
    "    # 模型预测\n",
    "    y_pred = gbm.predict_proba(test)[:, 1]  # predict_proba 返回n行k列的矩阵，第i行第j列上的数值是模型预测第i个预测样本为某个标签的概率, 这里的1表示点击的概率\n",
    "    print('predict: ', y_pred[:10]) # 这里看前10个， 预测为点击的概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:03:05.585613Z",
     "start_time": "2020-09-10T12:03:02.318820Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttrain's binary_logloss: 0.524173\tval's binary_logloss: 0.457618\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[2]\ttrain's binary_logloss: 0.521552\tval's binary_logloss: 0.457412\n",
      "[3]\ttrain's binary_logloss: 0.519105\tval's binary_logloss: 0.456826\n",
      "[4]\ttrain's binary_logloss: 0.516854\tval's binary_logloss: 0.45624\n",
      "[5]\ttrain's binary_logloss: 0.514682\tval's binary_logloss: 0.455686\n",
      "[6]\ttrain's binary_logloss: 0.512282\tval's binary_logloss: 0.455301\n",
      "[7]\ttrain's binary_logloss: 0.510138\tval's binary_logloss: 0.45498\n",
      "[8]\ttrain's binary_logloss: 0.507908\tval's binary_logloss: 0.454581\n",
      "[9]\ttrain's binary_logloss: 0.505686\tval's binary_logloss: 0.454076\n",
      "[10]\ttrain's binary_logloss: 0.503663\tval's binary_logloss: 0.454104\n",
      "[11]\ttrain's binary_logloss: 0.501517\tval's binary_logloss: 0.453702\n",
      "[12]\ttrain's binary_logloss: 0.499502\tval's binary_logloss: 0.45267\n",
      "[13]\ttrain's binary_logloss: 0.497524\tval's binary_logloss: 0.452138\n",
      "[14]\ttrain's binary_logloss: 0.495373\tval's binary_logloss: 0.452018\n",
      "[15]\ttrain's binary_logloss: 0.493334\tval's binary_logloss: 0.451436\n",
      "[16]\ttrain's binary_logloss: 0.491277\tval's binary_logloss: 0.450857\n",
      "[17]\ttrain's binary_logloss: 0.489399\tval's binary_logloss: 0.450653\n",
      "[18]\ttrain's binary_logloss: 0.487507\tval's binary_logloss: 0.450226\n",
      "[19]\ttrain's binary_logloss: 0.485694\tval's binary_logloss: 0.45037\n",
      "[20]\ttrain's binary_logloss: 0.483876\tval's binary_logloss: 0.44997\n",
      "[21]\ttrain's binary_logloss: 0.482046\tval's binary_logloss: 0.44965\n",
      "[22]\ttrain's binary_logloss: 0.480149\tval's binary_logloss: 0.449364\n",
      "[23]\ttrain's binary_logloss: 0.478524\tval's binary_logloss: 0.448982\n",
      "[24]\ttrain's binary_logloss: 0.476937\tval's binary_logloss: 0.44856\n",
      "[25]\ttrain's binary_logloss: 0.475253\tval's binary_logloss: 0.44839\n",
      "[26]\ttrain's binary_logloss: 0.473376\tval's binary_logloss: 0.447932\n",
      "[27]\ttrain's binary_logloss: 0.471746\tval's binary_logloss: 0.447652\n",
      "[28]\ttrain's binary_logloss: 0.469968\tval's binary_logloss: 0.447623\n",
      "[29]\ttrain's binary_logloss: 0.468318\tval's binary_logloss: 0.447119\n",
      "[30]\ttrain's binary_logloss: 0.466622\tval's binary_logloss: 0.446756\n",
      "[31]\ttrain's binary_logloss: 0.464789\tval's binary_logloss: 0.446734\n",
      "[32]\ttrain's binary_logloss: 0.463012\tval's binary_logloss: 0.446589\n",
      "[33]\ttrain's binary_logloss: 0.461385\tval's binary_logloss: 0.446432\n",
      "[34]\ttrain's binary_logloss: 0.45965\tval's binary_logloss: 0.445963\n",
      "[35]\ttrain's binary_logloss: 0.457976\tval's binary_logloss: 0.445911\n",
      "[36]\ttrain's binary_logloss: 0.456264\tval's binary_logloss: 0.445754\n",
      "[37]\ttrain's binary_logloss: 0.454581\tval's binary_logloss: 0.445687\n",
      "[38]\ttrain's binary_logloss: 0.452967\tval's binary_logloss: 0.445629\n",
      "[39]\ttrain's binary_logloss: 0.451271\tval's binary_logloss: 0.444983\n",
      "[40]\ttrain's binary_logloss: 0.449714\tval's binary_logloss: 0.444918\n",
      "[41]\ttrain's binary_logloss: 0.44821\tval's binary_logloss: 0.444414\n",
      "[42]\ttrain's binary_logloss: 0.446656\tval's binary_logloss: 0.444015\n",
      "[43]\ttrain's binary_logloss: 0.445142\tval's binary_logloss: 0.443646\n",
      "[44]\ttrain's binary_logloss: 0.443669\tval's binary_logloss: 0.443278\n",
      "[45]\ttrain's binary_logloss: 0.442152\tval's binary_logloss: 0.443147\n",
      "[46]\ttrain's binary_logloss: 0.440672\tval's binary_logloss: 0.442961\n",
      "[47]\ttrain's binary_logloss: 0.439199\tval's binary_logloss: 0.442584\n",
      "[48]\ttrain's binary_logloss: 0.437859\tval's binary_logloss: 0.442701\n",
      "[49]\ttrain's binary_logloss: 0.436464\tval's binary_logloss: 0.442506\n",
      "[50]\ttrain's binary_logloss: 0.435169\tval's binary_logloss: 0.442165\n",
      "[51]\ttrain's binary_logloss: 0.433832\tval's binary_logloss: 0.44216\n",
      "[52]\ttrain's binary_logloss: 0.432463\tval's binary_logloss: 0.441977\n",
      "[53]\ttrain's binary_logloss: 0.431202\tval's binary_logloss: 0.441679\n",
      "[54]\ttrain's binary_logloss: 0.429758\tval's binary_logloss: 0.44155\n",
      "[55]\ttrain's binary_logloss: 0.428491\tval's binary_logloss: 0.441271\n",
      "[56]\ttrain's binary_logloss: 0.427062\tval's binary_logloss: 0.44108\n",
      "[57]\ttrain's binary_logloss: 0.425698\tval's binary_logloss: 0.440738\n",
      "[58]\ttrain's binary_logloss: 0.424241\tval's binary_logloss: 0.440797\n",
      "[59]\ttrain's binary_logloss: 0.422834\tval's binary_logloss: 0.440581\n",
      "[60]\ttrain's binary_logloss: 0.421451\tval's binary_logloss: 0.44039\n",
      "[61]\ttrain's binary_logloss: 0.420355\tval's binary_logloss: 0.440308\n",
      "[62]\ttrain's binary_logloss: 0.419062\tval's binary_logloss: 0.440291\n",
      "[63]\ttrain's binary_logloss: 0.417794\tval's binary_logloss: 0.440425\n",
      "[64]\ttrain's binary_logloss: 0.416591\tval's binary_logloss: 0.440271\n",
      "[65]\ttrain's binary_logloss: 0.415303\tval's binary_logloss: 0.440208\n",
      "[66]\ttrain's binary_logloss: 0.413889\tval's binary_logloss: 0.439996\n",
      "[67]\ttrain's binary_logloss: 0.412497\tval's binary_logloss: 0.439695\n",
      "[68]\ttrain's binary_logloss: 0.411124\tval's binary_logloss: 0.439388\n",
      "[69]\ttrain's binary_logloss: 0.409791\tval's binary_logloss: 0.439328\n",
      "[70]\ttrain's binary_logloss: 0.408524\tval's binary_logloss: 0.439289\n",
      "[71]\ttrain's binary_logloss: 0.407282\tval's binary_logloss: 0.438865\n",
      "[72]\ttrain's binary_logloss: 0.406313\tval's binary_logloss: 0.43869\n",
      "[73]\ttrain's binary_logloss: 0.40511\tval's binary_logloss: 0.438465\n",
      "[74]\ttrain's binary_logloss: 0.403783\tval's binary_logloss: 0.438311\n",
      "[75]\ttrain's binary_logloss: 0.402595\tval's binary_logloss: 0.438056\n",
      "[76]\ttrain's binary_logloss: 0.401395\tval's binary_logloss: 0.438112\n",
      "[77]\ttrain's binary_logloss: 0.400291\tval's binary_logloss: 0.438035\n",
      "[78]\ttrain's binary_logloss: 0.399076\tval's binary_logloss: 0.438021\n",
      "[79]\ttrain's binary_logloss: 0.3978\tval's binary_logloss: 0.437625\n",
      "[80]\ttrain's binary_logloss: 0.396555\tval's binary_logloss: 0.437444\n",
      "[81]\ttrain's binary_logloss: 0.3955\tval's binary_logloss: 0.437557\n",
      "[82]\ttrain's binary_logloss: 0.394216\tval's binary_logloss: 0.437573\n",
      "[83]\ttrain's binary_logloss: 0.393026\tval's binary_logloss: 0.437414\n",
      "[84]\ttrain's binary_logloss: 0.391888\tval's binary_logloss: 0.437303\n",
      "[85]\ttrain's binary_logloss: 0.39081\tval's binary_logloss: 0.437463\n",
      "[86]\ttrain's binary_logloss: 0.389575\tval's binary_logloss: 0.437504\n",
      "[87]\ttrain's binary_logloss: 0.388486\tval's binary_logloss: 0.43758\n",
      "[88]\ttrain's binary_logloss: 0.387417\tval's binary_logloss: 0.437566\n",
      "[89]\ttrain's binary_logloss: 0.386285\tval's binary_logloss: 0.437373\n",
      "[90]\ttrain's binary_logloss: 0.385094\tval's binary_logloss: 0.437367\n",
      "[91]\ttrain's binary_logloss: 0.383948\tval's binary_logloss: 0.437307\n",
      "[92]\ttrain's binary_logloss: 0.382841\tval's binary_logloss: 0.437432\n",
      "[93]\ttrain's binary_logloss: 0.381775\tval's binary_logloss: 0.437536\n",
      "[94]\ttrain's binary_logloss: 0.380604\tval's binary_logloss: 0.437459\n",
      "[95]\ttrain's binary_logloss: 0.379311\tval's binary_logloss: 0.437236\n",
      "[96]\ttrain's binary_logloss: 0.378217\tval's binary_logloss: 0.436956\n",
      "[97]\ttrain's binary_logloss: 0.377272\tval's binary_logloss: 0.436941\n",
      "[98]\ttrain's binary_logloss: 0.37616\tval's binary_logloss: 0.436804\n",
      "[99]\ttrain's binary_logloss: 0.375124\tval's binary_logloss: 0.436772\n",
      "[100]\ttrain's binary_logloss: 0.374122\tval's binary_logloss: 0.436657\n",
      "[101]\ttrain's binary_logloss: 0.373153\tval's binary_logloss: 0.436627\n",
      "[102]\ttrain's binary_logloss: 0.372089\tval's binary_logloss: 0.436784\n",
      "[103]\ttrain's binary_logloss: 0.3711\tval's binary_logloss: 0.436692\n",
      "[104]\ttrain's binary_logloss: 0.370028\tval's binary_logloss: 0.436809\n",
      "[105]\ttrain's binary_logloss: 0.368918\tval's binary_logloss: 0.436842\n",
      "[106]\ttrain's binary_logloss: 0.367924\tval's binary_logloss: 0.437034\n",
      "[107]\ttrain's binary_logloss: 0.366838\tval's binary_logloss: 0.436775\n",
      "[108]\ttrain's binary_logloss: 0.365815\tval's binary_logloss: 0.436844\n",
      "[109]\ttrain's binary_logloss: 0.364815\tval's binary_logloss: 0.437027\n",
      "[110]\ttrain's binary_logloss: 0.363743\tval's binary_logloss: 0.436877\n",
      "[111]\ttrain's binary_logloss: 0.362978\tval's binary_logloss: 0.437046\n",
      "[112]\ttrain's binary_logloss: 0.362051\tval's binary_logloss: 0.437234\n",
      "[113]\ttrain's binary_logloss: 0.361091\tval's binary_logloss: 0.437297\n",
      "[114]\ttrain's binary_logloss: 0.360135\tval's binary_logloss: 0.437334\n",
      "[115]\ttrain's binary_logloss: 0.35927\tval's binary_logloss: 0.437215\n",
      "[116]\ttrain's binary_logloss: 0.358371\tval's binary_logloss: 0.43679\n",
      "[117]\ttrain's binary_logloss: 0.357374\tval's binary_logloss: 0.436852\n",
      "[118]\ttrain's binary_logloss: 0.356443\tval's binary_logloss: 0.436969\n",
      "[119]\ttrain's binary_logloss: 0.355366\tval's binary_logloss: 0.436644\n",
      "[120]\ttrain's binary_logloss: 0.3543\tval's binary_logloss: 0.436649\n",
      "[121]\ttrain's binary_logloss: 0.353574\tval's binary_logloss: 0.436735\n",
      "[122]\ttrain's binary_logloss: 0.352623\tval's binary_logloss: 0.436887\n",
      "[123]\ttrain's binary_logloss: 0.351634\tval's binary_logloss: 0.436616\n",
      "[124]\ttrain's binary_logloss: 0.350922\tval's binary_logloss: 0.436669\n",
      "[125]\ttrain's binary_logloss: 0.349962\tval's binary_logloss: 0.436485\n",
      "[126]\ttrain's binary_logloss: 0.349142\tval's binary_logloss: 0.436289\n",
      "[127]\ttrain's binary_logloss: 0.348405\tval's binary_logloss: 0.436282\n",
      "[128]\ttrain's binary_logloss: 0.34777\tval's binary_logloss: 0.436239\n",
      "[129]\ttrain's binary_logloss: 0.346828\tval's binary_logloss: 0.436232\n",
      "[130]\ttrain's binary_logloss: 0.346247\tval's binary_logloss: 0.436082\n",
      "[131]\ttrain's binary_logloss: 0.345473\tval's binary_logloss: 0.435985\n",
      "[132]\ttrain's binary_logloss: 0.345058\tval's binary_logloss: 0.435848\n",
      "[133]\ttrain's binary_logloss: 0.343993\tval's binary_logloss: 0.435684\n",
      "[134]\ttrain's binary_logloss: 0.342955\tval's binary_logloss: 0.435561\n",
      "[135]\ttrain's binary_logloss: 0.342074\tval's binary_logloss: 0.435341\n",
      "[136]\ttrain's binary_logloss: 0.341115\tval's binary_logloss: 0.435481\n",
      "[137]\ttrain's binary_logloss: 0.340603\tval's binary_logloss: 0.435421\n",
      "[138]\ttrain's binary_logloss: 0.339707\tval's binary_logloss: 0.435314\n",
      "[139]\ttrain's binary_logloss: 0.338767\tval's binary_logloss: 0.435247\n",
      "[140]\ttrain's binary_logloss: 0.337805\tval's binary_logloss: 0.435102\n",
      "[141]\ttrain's binary_logloss: 0.336942\tval's binary_logloss: 0.435408\n",
      "[142]\ttrain's binary_logloss: 0.336349\tval's binary_logloss: 0.435401\n",
      "[143]\ttrain's binary_logloss: 0.3354\tval's binary_logloss: 0.435405\n",
      "[144]\ttrain's binary_logloss: 0.334578\tval's binary_logloss: 0.435296\n",
      "[145]\ttrain's binary_logloss: 0.334011\tval's binary_logloss: 0.435274\n",
      "[146]\ttrain's binary_logloss: 0.333154\tval's binary_logloss: 0.435284\n",
      "[147]\ttrain's binary_logloss: 0.332569\tval's binary_logloss: 0.435335\n",
      "[148]\ttrain's binary_logloss: 0.331691\tval's binary_logloss: 0.435223\n",
      "[149]\ttrain's binary_logloss: 0.331119\tval's binary_logloss: 0.435246\n",
      "[150]\ttrain's binary_logloss: 0.330279\tval's binary_logloss: 0.435304\n",
      "[151]\ttrain's binary_logloss: 0.329611\tval's binary_logloss: 0.435434\n",
      "[152]\ttrain's binary_logloss: 0.328744\tval's binary_logloss: 0.435585\n",
      "[153]\ttrain's binary_logloss: 0.328271\tval's binary_logloss: 0.435756\n",
      "[154]\ttrain's binary_logloss: 0.327525\tval's binary_logloss: 0.435602\n",
      "[155]\ttrain's binary_logloss: 0.326977\tval's binary_logloss: 0.435631\n",
      "[156]\ttrain's binary_logloss: 0.326245\tval's binary_logloss: 0.435876\n",
      "[157]\ttrain's binary_logloss: 0.325416\tval's binary_logloss: 0.43598\n",
      "[158]\ttrain's binary_logloss: 0.32478\tval's binary_logloss: 0.435891\n",
      "[159]\ttrain's binary_logloss: 0.323879\tval's binary_logloss: 0.435996\n",
      "[160]\ttrain's binary_logloss: 0.323431\tval's binary_logloss: 0.435849\n",
      "[161]\ttrain's binary_logloss: 0.322469\tval's binary_logloss: 0.435929\n",
      "[162]\ttrain's binary_logloss: 0.321688\tval's binary_logloss: 0.435973\n",
      "[163]\ttrain's binary_logloss: 0.321027\tval's binary_logloss: 0.4361\n",
      "[164]\ttrain's binary_logloss: 0.320382\tval's binary_logloss: 0.436239\n",
      "[165]\ttrain's binary_logloss: 0.319819\tval's binary_logloss: 0.436498\n",
      "[166]\ttrain's binary_logloss: 0.318973\tval's binary_logloss: 0.436388\n",
      "[167]\ttrain's binary_logloss: 0.318217\tval's binary_logloss: 0.436108\n",
      "[168]\ttrain's binary_logloss: 0.317408\tval's binary_logloss: 0.436149\n",
      "[169]\ttrain's binary_logloss: 0.316683\tval's binary_logloss: 0.436128\n",
      "[170]\ttrain's binary_logloss: 0.31631\tval's binary_logloss: 0.436116\n",
      "[171]\ttrain's binary_logloss: 0.315546\tval's binary_logloss: 0.435937\n",
      "[172]\ttrain's binary_logloss: 0.315146\tval's binary_logloss: 0.435896\n",
      "[173]\ttrain's binary_logloss: 0.314369\tval's binary_logloss: 0.435878\n",
      "[174]\ttrain's binary_logloss: 0.313589\tval's binary_logloss: 0.4358\n",
      "[175]\ttrain's binary_logloss: 0.313241\tval's binary_logloss: 0.435738\n",
      "[176]\ttrain's binary_logloss: 0.31253\tval's binary_logloss: 0.435779\n",
      "[177]\ttrain's binary_logloss: 0.312195\tval's binary_logloss: 0.435769\n",
      "[178]\ttrain's binary_logloss: 0.311615\tval's binary_logloss: 0.435757\n",
      "[179]\ttrain's binary_logloss: 0.31094\tval's binary_logloss: 0.435736\n",
      "[180]\ttrain's binary_logloss: 0.31028\tval's binary_logloss: 0.435736\n",
      "[181]\ttrain's binary_logloss: 0.309393\tval's binary_logloss: 0.435816\n",
      "[182]\ttrain's binary_logloss: 0.308739\tval's binary_logloss: 0.435773\n",
      "[183]\ttrain's binary_logloss: 0.308052\tval's binary_logloss: 0.435788\n",
      "[184]\ttrain's binary_logloss: 0.307147\tval's binary_logloss: 0.435754\n",
      "[185]\ttrain's binary_logloss: 0.306299\tval's binary_logloss: 0.43588\n",
      "[186]\ttrain's binary_logloss: 0.305971\tval's binary_logloss: 0.435835\n",
      "[187]\ttrain's binary_logloss: 0.305252\tval's binary_logloss: 0.436017\n",
      "[188]\ttrain's binary_logloss: 0.304524\tval's binary_logloss: 0.436035\n",
      "[189]\ttrain's binary_logloss: 0.304231\tval's binary_logloss: 0.436063\n",
      "[190]\ttrain's binary_logloss: 0.303511\tval's binary_logloss: 0.436016\n",
      "[191]\ttrain's binary_logloss: 0.30281\tval's binary_logloss: 0.436093\n",
      "[192]\ttrain's binary_logloss: 0.302154\tval's binary_logloss: 0.436288\n",
      "[193]\ttrain's binary_logloss: 0.301454\tval's binary_logloss: 0.43619\n",
      "[194]\ttrain's binary_logloss: 0.300786\tval's binary_logloss: 0.436296\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[195]\ttrain's binary_logloss: 0.30035\tval's binary_logloss: 0.4364\n",
      "[196]\ttrain's binary_logloss: 0.299616\tval's binary_logloss: 0.436411\n",
      "[197]\ttrain's binary_logloss: 0.298919\tval's binary_logloss: 0.436513\n",
      "[198]\ttrain's binary_logloss: 0.298072\tval's binary_logloss: 0.436658\n",
      "[199]\ttrain's binary_logloss: 0.297682\tval's binary_logloss: 0.43674\n",
      "[200]\ttrain's binary_logloss: 0.296978\tval's binary_logloss: 0.437032\n",
      "[201]\ttrain's binary_logloss: 0.29671\tval's binary_logloss: 0.437081\n",
      "[202]\ttrain's binary_logloss: 0.295998\tval's binary_logloss: 0.437004\n",
      "[203]\ttrain's binary_logloss: 0.295393\tval's binary_logloss: 0.436971\n",
      "[204]\ttrain's binary_logloss: 0.294671\tval's binary_logloss: 0.437022\n",
      "[205]\ttrain's binary_logloss: 0.293954\tval's binary_logloss: 0.437149\n",
      "[206]\ttrain's binary_logloss: 0.293231\tval's binary_logloss: 0.437328\n",
      "[207]\ttrain's binary_logloss: 0.292771\tval's binary_logloss: 0.437319\n",
      "[208]\ttrain's binary_logloss: 0.292246\tval's binary_logloss: 0.437314\n",
      "[209]\ttrain's binary_logloss: 0.291532\tval's binary_logloss: 0.437209\n",
      "[210]\ttrain's binary_logloss: 0.29085\tval's binary_logloss: 0.437075\n",
      "[211]\ttrain's binary_logloss: 0.290135\tval's binary_logloss: 0.437083\n",
      "[212]\ttrain's binary_logloss: 0.289453\tval's binary_logloss: 0.436936\n",
      "[213]\ttrain's binary_logloss: 0.288979\tval's binary_logloss: 0.436909\n",
      "[214]\ttrain's binary_logloss: 0.288446\tval's binary_logloss: 0.436968\n",
      "[215]\ttrain's binary_logloss: 0.287758\tval's binary_logloss: 0.437057\n",
      "[216]\ttrain's binary_logloss: 0.287193\tval's binary_logloss: 0.436949\n",
      "[217]\ttrain's binary_logloss: 0.286627\tval's binary_logloss: 0.436935\n",
      "[218]\ttrain's binary_logloss: 0.285894\tval's binary_logloss: 0.437198\n",
      "[219]\ttrain's binary_logloss: 0.285213\tval's binary_logloss: 0.437363\n",
      "[220]\ttrain's binary_logloss: 0.28457\tval's binary_logloss: 0.43745\n",
      "[221]\ttrain's binary_logloss: 0.284015\tval's binary_logloss: 0.437494\n",
      "[222]\ttrain's binary_logloss: 0.283267\tval's binary_logloss: 0.437576\n",
      "[223]\ttrain's binary_logloss: 0.282604\tval's binary_logloss: 0.437545\n",
      "[224]\ttrain's binary_logloss: 0.282281\tval's binary_logloss: 0.437651\n",
      "[225]\ttrain's binary_logloss: 0.281528\tval's binary_logloss: 0.437425\n",
      "[226]\ttrain's binary_logloss: 0.280983\tval's binary_logloss: 0.437208\n",
      "[227]\ttrain's binary_logloss: 0.280241\tval's binary_logloss: 0.437625\n",
      "[228]\ttrain's binary_logloss: 0.279592\tval's binary_logloss: 0.437633\n",
      "[229]\ttrain's binary_logloss: 0.279303\tval's binary_logloss: 0.437702\n",
      "[230]\ttrain's binary_logloss: 0.278863\tval's binary_logloss: 0.437728\n",
      "[231]\ttrain's binary_logloss: 0.278114\tval's binary_logloss: 0.437751\n",
      "[232]\ttrain's binary_logloss: 0.277583\tval's binary_logloss: 0.437956\n",
      "[233]\ttrain's binary_logloss: 0.277318\tval's binary_logloss: 0.438045\n",
      "[234]\ttrain's binary_logloss: 0.276707\tval's binary_logloss: 0.437967\n",
      "[235]\ttrain's binary_logloss: 0.2761\tval's binary_logloss: 0.438425\n",
      "[236]\ttrain's binary_logloss: 0.275484\tval's binary_logloss: 0.438495\n",
      "[237]\ttrain's binary_logloss: 0.275111\tval's binary_logloss: 0.438675\n",
      "[238]\ttrain's binary_logloss: 0.274511\tval's binary_logloss: 0.438652\n",
      "[239]\ttrain's binary_logloss: 0.273894\tval's binary_logloss: 0.438641\n",
      "[240]\ttrain's binary_logloss: 0.273434\tval's binary_logloss: 0.438684\n",
      "Early stopping, best iteration is:\n",
      "[140]\ttrain's binary_logloss: 0.337805\tval's binary_logloss: 0.435102\n",
      "tr_logloss:  0.33780468134640934\n",
      "val_logloss:  0.43510196864973577\n",
      "predict:  [0.41085041 0.34548338 0.23243039 0.14776386 0.17582282 0.36713449\n",
      " 0.18294486 0.11630654 0.10178178 0.3018972 ]\n"
     ]
    }
   ],
   "source": [
    "# 模型训练和预测\n",
    "gbdt_model(data.copy(), category_fea, continuous_fea)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LR + GBDT建模\n",
    "下面就是把上面两个模型进行组合， GBDT负责对各个特征进行交叉和组合， 把原始特征向量转换为新的离散型特征向量， 然后在使用逻辑回归模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:08:05.780599Z",
     "start_time": "2020-09-10T12:08:05.765639Z"
    }
   },
   "outputs": [],
   "source": [
    "def gbdt_lr_model(data, category_feature, continuous_feature): # 0.43616\n",
    "    # 离散特征one-hot编码\n",
    "    for col in category_feature:\n",
    "        onehot_feats = pd.get_dummies(data[col], prefix = col)\n",
    "        data.drop([col], axis = 1, inplace = True)\n",
    "        data = pd.concat([data, onehot_feats], axis = 1)\n",
    "\n",
    "    train = data[data['Label'] != -1]\n",
    "    target = train.pop('Label')\n",
    "    test = data[data['Label'] == -1]\n",
    "    test.drop(['Label'], axis = 1, inplace = True)\n",
    "\n",
    "    # 划分数据集\n",
    "    x_train, x_val, y_train, y_val = train_test_split(train, target, test_size = 0.2, random_state = 2020)\n",
    "\n",
    "    gbm = lgb.LGBMClassifier(objective='binary',\n",
    "                            subsample= 0.8,\n",
    "                            min_child_weight= 0.5,\n",
    "                            colsample_bytree= 0.7,\n",
    "                            num_leaves=100,\n",
    "                            max_depth = 12,\n",
    "                            learning_rate=0.01,\n",
    "                            n_estimators=1000,\n",
    "                            )\n",
    "\n",
    "    gbm.fit(x_train, y_train,\n",
    "            eval_set = [(x_train, y_train), (x_val, y_val)],\n",
    "            eval_names = ['train', 'val'],\n",
    "            eval_metric = 'binary_logloss',\n",
    "            early_stopping_rounds = 100,\n",
    "            )\n",
    "    \n",
    "    model = gbm.booster_\n",
    "\n",
    "    gbdt_feats_train = model.predict(train, pred_leaf = True)\n",
    "    gbdt_feats_test = model.predict(test, pred_leaf = True)\n",
    "    gbdt_feats_name = ['gbdt_leaf_' + str(i) for i in range(gbdt_feats_train.shape[1])]\n",
    "    df_train_gbdt_feats = pd.DataFrame(gbdt_feats_train, columns = gbdt_feats_name) \n",
    "    df_test_gbdt_feats = pd.DataFrame(gbdt_feats_test, columns = gbdt_feats_name)\n",
    "\n",
    "    train = pd.concat([train, df_train_gbdt_feats], axis = 1)\n",
    "    test = pd.concat([test, df_test_gbdt_feats], axis = 1)\n",
    "    train_len = train.shape[0]\n",
    "    data = pd.concat([train, test])\n",
    "    del train\n",
    "    del test\n",
    "    gc.collect()\n",
    "\n",
    "    # # 连续特征归一化\n",
    "    scaler = MinMaxScaler()\n",
    "    for col in continuous_feature:\n",
    "        data[col] = scaler.fit_transform(data[col].values.reshape(-1, 1))\n",
    "\n",
    "    for col in gbdt_feats_name:\n",
    "        onehot_feats = pd.get_dummies(data[col], prefix = col)\n",
    "        data.drop([col], axis = 1, inplace = True)\n",
    "        data = pd.concat([data, onehot_feats], axis = 1)\n",
    "\n",
    "    train = data[: train_len]\n",
    "    test = data[train_len:]\n",
    "    del data\n",
    "    gc.collect()\n",
    "\n",
    "    x_train, x_val, y_train, y_val = train_test_split(train, target, test_size = 0.3, random_state = 2018)\n",
    "\n",
    "    \n",
    "    lr = LogisticRegression()\n",
    "    lr.fit(x_train, y_train)\n",
    "    tr_logloss = log_loss(y_train, lr.predict_proba(x_train)[:, 1])\n",
    "    print('tr-logloss: ', tr_logloss)\n",
    "    val_logloss = log_loss(y_val, lr.predict_proba(x_val)[:, 1])\n",
    "    print('val-logloss: ', val_logloss)\n",
    "    y_pred = lr.predict_proba(test)[:, 1]\n",
    "    print(y_pred[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-10T12:08:32.439103Z",
     "start_time": "2020-09-10T12:08:07.548420Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "开始one-hot...\n",
      "one-hot结束\n",
      "划分数据集...\n",
      "开始训练gbdt..\n",
      "[1]\ttrain's binary_logloss: 0.524173\tval's binary_logloss: 0.457618\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[2]\ttrain's binary_logloss: 0.521552\tval's binary_logloss: 0.457412\n",
      "[3]\ttrain's binary_logloss: 0.519105\tval's binary_logloss: 0.456826\n",
      "[4]\ttrain's binary_logloss: 0.516854\tval's binary_logloss: 0.45624\n",
      "[5]\ttrain's binary_logloss: 0.514682\tval's binary_logloss: 0.455686\n",
      "[6]\ttrain's binary_logloss: 0.512282\tval's binary_logloss: 0.455301\n",
      "[7]\ttrain's binary_logloss: 0.510138\tval's binary_logloss: 0.45498\n",
      "[8]\ttrain's binary_logloss: 0.507908\tval's binary_logloss: 0.454581\n",
      "[9]\ttrain's binary_logloss: 0.505686\tval's binary_logloss: 0.454076\n",
      "[10]\ttrain's binary_logloss: 0.503663\tval's binary_logloss: 0.454104\n",
      "[11]\ttrain's binary_logloss: 0.501517\tval's binary_logloss: 0.453702\n",
      "[12]\ttrain's binary_logloss: 0.499502\tval's binary_logloss: 0.45267\n",
      "[13]\ttrain's binary_logloss: 0.497524\tval's binary_logloss: 0.452138\n",
      "[14]\ttrain's binary_logloss: 0.495373\tval's binary_logloss: 0.452018\n",
      "[15]\ttrain's binary_logloss: 0.493334\tval's binary_logloss: 0.451436\n",
      "[16]\ttrain's binary_logloss: 0.491277\tval's binary_logloss: 0.450857\n",
      "[17]\ttrain's binary_logloss: 0.489399\tval's binary_logloss: 0.450653\n",
      "[18]\ttrain's binary_logloss: 0.487507\tval's binary_logloss: 0.450226\n",
      "[19]\ttrain's binary_logloss: 0.485694\tval's binary_logloss: 0.45037\n",
      "[20]\ttrain's binary_logloss: 0.483876\tval's binary_logloss: 0.44997\n",
      "[21]\ttrain's binary_logloss: 0.482046\tval's binary_logloss: 0.44965\n",
      "[22]\ttrain's binary_logloss: 0.480149\tval's binary_logloss: 0.449364\n",
      "[23]\ttrain's binary_logloss: 0.478524\tval's binary_logloss: 0.448982\n",
      "[24]\ttrain's binary_logloss: 0.476937\tval's binary_logloss: 0.44856\n",
      "[25]\ttrain's binary_logloss: 0.475253\tval's binary_logloss: 0.44839\n",
      "[26]\ttrain's binary_logloss: 0.473376\tval's binary_logloss: 0.447932\n",
      "[27]\ttrain's binary_logloss: 0.471746\tval's binary_logloss: 0.447652\n",
      "[28]\ttrain's binary_logloss: 0.469968\tval's binary_logloss: 0.447623\n",
      "[29]\ttrain's binary_logloss: 0.468318\tval's binary_logloss: 0.447119\n",
      "[30]\ttrain's binary_logloss: 0.466622\tval's binary_logloss: 0.446756\n",
      "[31]\ttrain's binary_logloss: 0.464789\tval's binary_logloss: 0.446734\n",
      "[32]\ttrain's binary_logloss: 0.463012\tval's binary_logloss: 0.446589\n",
      "[33]\ttrain's binary_logloss: 0.461385\tval's binary_logloss: 0.446432\n",
      "[34]\ttrain's binary_logloss: 0.45965\tval's binary_logloss: 0.445963\n",
      "[35]\ttrain's binary_logloss: 0.457976\tval's binary_logloss: 0.445911\n",
      "[36]\ttrain's binary_logloss: 0.456264\tval's binary_logloss: 0.445754\n",
      "[37]\ttrain's binary_logloss: 0.454581\tval's binary_logloss: 0.445687\n",
      "[38]\ttrain's binary_logloss: 0.452967\tval's binary_logloss: 0.445629\n",
      "[39]\ttrain's binary_logloss: 0.451271\tval's binary_logloss: 0.444983\n",
      "[40]\ttrain's binary_logloss: 0.449714\tval's binary_logloss: 0.444918\n",
      "[41]\ttrain's binary_logloss: 0.44821\tval's binary_logloss: 0.444414\n",
      "[42]\ttrain's binary_logloss: 0.446656\tval's binary_logloss: 0.444015\n",
      "[43]\ttrain's binary_logloss: 0.445142\tval's binary_logloss: 0.443646\n",
      "[44]\ttrain's binary_logloss: 0.443669\tval's binary_logloss: 0.443278\n",
      "[45]\ttrain's binary_logloss: 0.442152\tval's binary_logloss: 0.443147\n",
      "[46]\ttrain's binary_logloss: 0.440672\tval's binary_logloss: 0.442961\n",
      "[47]\ttrain's binary_logloss: 0.439199\tval's binary_logloss: 0.442584\n",
      "[48]\ttrain's binary_logloss: 0.437859\tval's binary_logloss: 0.442701\n",
      "[49]\ttrain's binary_logloss: 0.436464\tval's binary_logloss: 0.442506\n",
      "[50]\ttrain's binary_logloss: 0.435169\tval's binary_logloss: 0.442165\n",
      "[51]\ttrain's binary_logloss: 0.433832\tval's binary_logloss: 0.44216\n",
      "[52]\ttrain's binary_logloss: 0.432463\tval's binary_logloss: 0.441977\n",
      "[53]\ttrain's binary_logloss: 0.431202\tval's binary_logloss: 0.441679\n",
      "[54]\ttrain's binary_logloss: 0.429758\tval's binary_logloss: 0.44155\n",
      "[55]\ttrain's binary_logloss: 0.428491\tval's binary_logloss: 0.441271\n",
      "[56]\ttrain's binary_logloss: 0.427062\tval's binary_logloss: 0.44108\n",
      "[57]\ttrain's binary_logloss: 0.425698\tval's binary_logloss: 0.440738\n",
      "[58]\ttrain's binary_logloss: 0.424241\tval's binary_logloss: 0.440797\n",
      "[59]\ttrain's binary_logloss: 0.422834\tval's binary_logloss: 0.440581\n",
      "[60]\ttrain's binary_logloss: 0.421451\tval's binary_logloss: 0.44039\n",
      "[61]\ttrain's binary_logloss: 0.420355\tval's binary_logloss: 0.440308\n",
      "[62]\ttrain's binary_logloss: 0.419062\tval's binary_logloss: 0.440291\n",
      "[63]\ttrain's binary_logloss: 0.417794\tval's binary_logloss: 0.440425\n",
      "[64]\ttrain's binary_logloss: 0.416591\tval's binary_logloss: 0.440271\n",
      "[65]\ttrain's binary_logloss: 0.415303\tval's binary_logloss: 0.440208\n",
      "[66]\ttrain's binary_logloss: 0.413889\tval's binary_logloss: 0.439996\n",
      "[67]\ttrain's binary_logloss: 0.412497\tval's binary_logloss: 0.439695\n",
      "[68]\ttrain's binary_logloss: 0.411124\tval's binary_logloss: 0.439388\n",
      "[69]\ttrain's binary_logloss: 0.409791\tval's binary_logloss: 0.439328\n",
      "[70]\ttrain's binary_logloss: 0.408524\tval's binary_logloss: 0.439289\n",
      "[71]\ttrain's binary_logloss: 0.407282\tval's binary_logloss: 0.438865\n",
      "[72]\ttrain's binary_logloss: 0.406313\tval's binary_logloss: 0.43869\n",
      "[73]\ttrain's binary_logloss: 0.40511\tval's binary_logloss: 0.438465\n",
      "[74]\ttrain's binary_logloss: 0.403783\tval's binary_logloss: 0.438311\n",
      "[75]\ttrain's binary_logloss: 0.402595\tval's binary_logloss: 0.438056\n",
      "[76]\ttrain's binary_logloss: 0.401395\tval's binary_logloss: 0.438112\n",
      "[77]\ttrain's binary_logloss: 0.400291\tval's binary_logloss: 0.438035\n",
      "[78]\ttrain's binary_logloss: 0.399076\tval's binary_logloss: 0.438021\n",
      "[79]\ttrain's binary_logloss: 0.3978\tval's binary_logloss: 0.437625\n",
      "[80]\ttrain's binary_logloss: 0.396555\tval's binary_logloss: 0.437444\n",
      "[81]\ttrain's binary_logloss: 0.3955\tval's binary_logloss: 0.437557\n",
      "[82]\ttrain's binary_logloss: 0.394216\tval's binary_logloss: 0.437573\n",
      "[83]\ttrain's binary_logloss: 0.393026\tval's binary_logloss: 0.437414\n",
      "[84]\ttrain's binary_logloss: 0.391888\tval's binary_logloss: 0.437303\n",
      "[85]\ttrain's binary_logloss: 0.39081\tval's binary_logloss: 0.437463\n",
      "[86]\ttrain's binary_logloss: 0.389575\tval's binary_logloss: 0.437504\n",
      "[87]\ttrain's binary_logloss: 0.388486\tval's binary_logloss: 0.43758\n",
      "[88]\ttrain's binary_logloss: 0.387417\tval's binary_logloss: 0.437566\n",
      "[89]\ttrain's binary_logloss: 0.386285\tval's binary_logloss: 0.437373\n",
      "[90]\ttrain's binary_logloss: 0.385094\tval's binary_logloss: 0.437367\n",
      "[91]\ttrain's binary_logloss: 0.383948\tval's binary_logloss: 0.437307\n",
      "[92]\ttrain's binary_logloss: 0.382841\tval's binary_logloss: 0.437432\n",
      "[93]\ttrain's binary_logloss: 0.381775\tval's binary_logloss: 0.437536\n",
      "[94]\ttrain's binary_logloss: 0.380604\tval's binary_logloss: 0.437459\n",
      "[95]\ttrain's binary_logloss: 0.379311\tval's binary_logloss: 0.437236\n",
      "[96]\ttrain's binary_logloss: 0.378217\tval's binary_logloss: 0.436956\n",
      "[97]\ttrain's binary_logloss: 0.377272\tval's binary_logloss: 0.436941\n",
      "[98]\ttrain's binary_logloss: 0.37616\tval's binary_logloss: 0.436804\n",
      "[99]\ttrain's binary_logloss: 0.375124\tval's binary_logloss: 0.436772\n",
      "[100]\ttrain's binary_logloss: 0.374122\tval's binary_logloss: 0.436657\n",
      "[101]\ttrain's binary_logloss: 0.373153\tval's binary_logloss: 0.436627\n",
      "[102]\ttrain's binary_logloss: 0.372089\tval's binary_logloss: 0.436784\n",
      "[103]\ttrain's binary_logloss: 0.3711\tval's binary_logloss: 0.436692\n",
      "[104]\ttrain's binary_logloss: 0.370028\tval's binary_logloss: 0.436809\n",
      "[105]\ttrain's binary_logloss: 0.368918\tval's binary_logloss: 0.436842\n",
      "[106]\ttrain's binary_logloss: 0.367924\tval's binary_logloss: 0.437034\n",
      "[107]\ttrain's binary_logloss: 0.366838\tval's binary_logloss: 0.436775\n",
      "[108]\ttrain's binary_logloss: 0.365815\tval's binary_logloss: 0.436844\n",
      "[109]\ttrain's binary_logloss: 0.364815\tval's binary_logloss: 0.437027\n",
      "[110]\ttrain's binary_logloss: 0.363743\tval's binary_logloss: 0.436877\n",
      "[111]\ttrain's binary_logloss: 0.362978\tval's binary_logloss: 0.437046\n",
      "[112]\ttrain's binary_logloss: 0.362051\tval's binary_logloss: 0.437234\n",
      "[113]\ttrain's binary_logloss: 0.361091\tval's binary_logloss: 0.437297\n",
      "[114]\ttrain's binary_logloss: 0.360135\tval's binary_logloss: 0.437334\n",
      "[115]\ttrain's binary_logloss: 0.35927\tval's binary_logloss: 0.437215\n",
      "[116]\ttrain's binary_logloss: 0.358371\tval's binary_logloss: 0.43679\n",
      "[117]\ttrain's binary_logloss: 0.357374\tval's binary_logloss: 0.436852\n",
      "[118]\ttrain's binary_logloss: 0.356443\tval's binary_logloss: 0.436969\n",
      "[119]\ttrain's binary_logloss: 0.355366\tval's binary_logloss: 0.436644\n",
      "[120]\ttrain's binary_logloss: 0.3543\tval's binary_logloss: 0.436649\n",
      "[121]\ttrain's binary_logloss: 0.353574\tval's binary_logloss: 0.436735\n",
      "[122]\ttrain's binary_logloss: 0.352623\tval's binary_logloss: 0.436887\n",
      "[123]\ttrain's binary_logloss: 0.351634\tval's binary_logloss: 0.436616\n",
      "[124]\ttrain's binary_logloss: 0.350922\tval's binary_logloss: 0.436669\n",
      "[125]\ttrain's binary_logloss: 0.349962\tval's binary_logloss: 0.436485\n",
      "[126]\ttrain's binary_logloss: 0.349142\tval's binary_logloss: 0.436289\n",
      "[127]\ttrain's binary_logloss: 0.348405\tval's binary_logloss: 0.436282\n",
      "[128]\ttrain's binary_logloss: 0.34777\tval's binary_logloss: 0.436239\n",
      "[129]\ttrain's binary_logloss: 0.346828\tval's binary_logloss: 0.436232\n",
      "[130]\ttrain's binary_logloss: 0.346247\tval's binary_logloss: 0.436082\n",
      "[131]\ttrain's binary_logloss: 0.345473\tval's binary_logloss: 0.435985\n",
      "[132]\ttrain's binary_logloss: 0.345058\tval's binary_logloss: 0.435848\n",
      "[133]\ttrain's binary_logloss: 0.343993\tval's binary_logloss: 0.435684\n",
      "[134]\ttrain's binary_logloss: 0.342955\tval's binary_logloss: 0.435561\n",
      "[135]\ttrain's binary_logloss: 0.342074\tval's binary_logloss: 0.435341\n",
      "[136]\ttrain's binary_logloss: 0.341115\tval's binary_logloss: 0.435481\n",
      "[137]\ttrain's binary_logloss: 0.340603\tval's binary_logloss: 0.435421\n",
      "[138]\ttrain's binary_logloss: 0.339707\tval's binary_logloss: 0.435314\n",
      "[139]\ttrain's binary_logloss: 0.338767\tval's binary_logloss: 0.435247\n",
      "[140]\ttrain's binary_logloss: 0.337805\tval's binary_logloss: 0.435102\n",
      "[141]\ttrain's binary_logloss: 0.336942\tval's binary_logloss: 0.435408\n",
      "[142]\ttrain's binary_logloss: 0.336349\tval's binary_logloss: 0.435401\n",
      "[143]\ttrain's binary_logloss: 0.3354\tval's binary_logloss: 0.435405\n",
      "[144]\ttrain's binary_logloss: 0.334578\tval's binary_logloss: 0.435296\n",
      "[145]\ttrain's binary_logloss: 0.334011\tval's binary_logloss: 0.435274\n",
      "[146]\ttrain's binary_logloss: 0.333154\tval's binary_logloss: 0.435284\n",
      "[147]\ttrain's binary_logloss: 0.332569\tval's binary_logloss: 0.435335\n",
      "[148]\ttrain's binary_logloss: 0.331691\tval's binary_logloss: 0.435223\n",
      "[149]\ttrain's binary_logloss: 0.331119\tval's binary_logloss: 0.435246\n",
      "[150]\ttrain's binary_logloss: 0.330279\tval's binary_logloss: 0.435304\n",
      "[151]\ttrain's binary_logloss: 0.329611\tval's binary_logloss: 0.435434\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[152]\ttrain's binary_logloss: 0.328744\tval's binary_logloss: 0.435585\n",
      "[153]\ttrain's binary_logloss: 0.328271\tval's binary_logloss: 0.435756\n",
      "[154]\ttrain's binary_logloss: 0.327525\tval's binary_logloss: 0.435602\n",
      "[155]\ttrain's binary_logloss: 0.326977\tval's binary_logloss: 0.435631\n",
      "[156]\ttrain's binary_logloss: 0.326245\tval's binary_logloss: 0.435876\n",
      "[157]\ttrain's binary_logloss: 0.325416\tval's binary_logloss: 0.43598\n",
      "[158]\ttrain's binary_logloss: 0.32478\tval's binary_logloss: 0.435891\n",
      "[159]\ttrain's binary_logloss: 0.323879\tval's binary_logloss: 0.435996\n",
      "[160]\ttrain's binary_logloss: 0.323431\tval's binary_logloss: 0.435849\n",
      "[161]\ttrain's binary_logloss: 0.322469\tval's binary_logloss: 0.435929\n",
      "[162]\ttrain's binary_logloss: 0.321688\tval's binary_logloss: 0.435973\n",
      "[163]\ttrain's binary_logloss: 0.321027\tval's binary_logloss: 0.4361\n",
      "[164]\ttrain's binary_logloss: 0.320382\tval's binary_logloss: 0.436239\n",
      "[165]\ttrain's binary_logloss: 0.319819\tval's binary_logloss: 0.436498\n",
      "[166]\ttrain's binary_logloss: 0.318973\tval's binary_logloss: 0.436388\n",
      "[167]\ttrain's binary_logloss: 0.318217\tval's binary_logloss: 0.436108\n",
      "[168]\ttrain's binary_logloss: 0.317408\tval's binary_logloss: 0.436149\n",
      "[169]\ttrain's binary_logloss: 0.316683\tval's binary_logloss: 0.436128\n",
      "[170]\ttrain's binary_logloss: 0.31631\tval's binary_logloss: 0.436116\n",
      "[171]\ttrain's binary_logloss: 0.315546\tval's binary_logloss: 0.435937\n",
      "[172]\ttrain's binary_logloss: 0.315146\tval's binary_logloss: 0.435896\n",
      "[173]\ttrain's binary_logloss: 0.314369\tval's binary_logloss: 0.435878\n",
      "[174]\ttrain's binary_logloss: 0.313589\tval's binary_logloss: 0.4358\n",
      "[175]\ttrain's binary_logloss: 0.313241\tval's binary_logloss: 0.435738\n",
      "[176]\ttrain's binary_logloss: 0.31253\tval's binary_logloss: 0.435779\n",
      "[177]\ttrain's binary_logloss: 0.312195\tval's binary_logloss: 0.435769\n",
      "[178]\ttrain's binary_logloss: 0.311615\tval's binary_logloss: 0.435757\n",
      "[179]\ttrain's binary_logloss: 0.31094\tval's binary_logloss: 0.435736\n",
      "[180]\ttrain's binary_logloss: 0.31028\tval's binary_logloss: 0.435736\n",
      "[181]\ttrain's binary_logloss: 0.309393\tval's binary_logloss: 0.435816\n",
      "[182]\ttrain's binary_logloss: 0.308739\tval's binary_logloss: 0.435773\n",
      "[183]\ttrain's binary_logloss: 0.308052\tval's binary_logloss: 0.435788\n",
      "[184]\ttrain's binary_logloss: 0.307147\tval's binary_logloss: 0.435754\n",
      "[185]\ttrain's binary_logloss: 0.306299\tval's binary_logloss: 0.43588\n",
      "[186]\ttrain's binary_logloss: 0.305971\tval's binary_logloss: 0.435835\n",
      "[187]\ttrain's binary_logloss: 0.305252\tval's binary_logloss: 0.436017\n",
      "[188]\ttrain's binary_logloss: 0.304524\tval's binary_logloss: 0.436035\n",
      "[189]\ttrain's binary_logloss: 0.304231\tval's binary_logloss: 0.436063\n",
      "[190]\ttrain's binary_logloss: 0.303511\tval's binary_logloss: 0.436016\n",
      "[191]\ttrain's binary_logloss: 0.30281\tval's binary_logloss: 0.436093\n",
      "[192]\ttrain's binary_logloss: 0.302154\tval's binary_logloss: 0.436288\n",
      "[193]\ttrain's binary_logloss: 0.301454\tval's binary_logloss: 0.43619\n",
      "[194]\ttrain's binary_logloss: 0.300786\tval's binary_logloss: 0.436296\n",
      "[195]\ttrain's binary_logloss: 0.30035\tval's binary_logloss: 0.4364\n",
      "[196]\ttrain's binary_logloss: 0.299616\tval's binary_logloss: 0.436411\n",
      "[197]\ttrain's binary_logloss: 0.298919\tval's binary_logloss: 0.436513\n",
      "[198]\ttrain's binary_logloss: 0.298072\tval's binary_logloss: 0.436658\n",
      "[199]\ttrain's binary_logloss: 0.297682\tval's binary_logloss: 0.43674\n",
      "[200]\ttrain's binary_logloss: 0.296978\tval's binary_logloss: 0.437032\n",
      "[201]\ttrain's binary_logloss: 0.29671\tval's binary_logloss: 0.437081\n",
      "[202]\ttrain's binary_logloss: 0.295998\tval's binary_logloss: 0.437004\n",
      "[203]\ttrain's binary_logloss: 0.295393\tval's binary_logloss: 0.436971\n",
      "[204]\ttrain's binary_logloss: 0.294671\tval's binary_logloss: 0.437022\n",
      "[205]\ttrain's binary_logloss: 0.293954\tval's binary_logloss: 0.437149\n",
      "[206]\ttrain's binary_logloss: 0.293231\tval's binary_logloss: 0.437328\n",
      "[207]\ttrain's binary_logloss: 0.292771\tval's binary_logloss: 0.437319\n",
      "[208]\ttrain's binary_logloss: 0.292246\tval's binary_logloss: 0.437314\n",
      "[209]\ttrain's binary_logloss: 0.291532\tval's binary_logloss: 0.437209\n",
      "[210]\ttrain's binary_logloss: 0.29085\tval's binary_logloss: 0.437075\n",
      "[211]\ttrain's binary_logloss: 0.290135\tval's binary_logloss: 0.437083\n",
      "[212]\ttrain's binary_logloss: 0.289453\tval's binary_logloss: 0.436936\n",
      "[213]\ttrain's binary_logloss: 0.288979\tval's binary_logloss: 0.436909\n",
      "[214]\ttrain's binary_logloss: 0.288446\tval's binary_logloss: 0.436968\n",
      "[215]\ttrain's binary_logloss: 0.287758\tval's binary_logloss: 0.437057\n",
      "[216]\ttrain's binary_logloss: 0.287193\tval's binary_logloss: 0.436949\n",
      "[217]\ttrain's binary_logloss: 0.286627\tval's binary_logloss: 0.436935\n",
      "[218]\ttrain's binary_logloss: 0.285894\tval's binary_logloss: 0.437198\n",
      "[219]\ttrain's binary_logloss: 0.285213\tval's binary_logloss: 0.437363\n",
      "[220]\ttrain's binary_logloss: 0.28457\tval's binary_logloss: 0.43745\n",
      "[221]\ttrain's binary_logloss: 0.284015\tval's binary_logloss: 0.437494\n",
      "[222]\ttrain's binary_logloss: 0.283267\tval's binary_logloss: 0.437576\n",
      "[223]\ttrain's binary_logloss: 0.282604\tval's binary_logloss: 0.437545\n",
      "[224]\ttrain's binary_logloss: 0.282281\tval's binary_logloss: 0.437651\n",
      "[225]\ttrain's binary_logloss: 0.281528\tval's binary_logloss: 0.437425\n",
      "[226]\ttrain's binary_logloss: 0.280983\tval's binary_logloss: 0.437208\n",
      "[227]\ttrain's binary_logloss: 0.280241\tval's binary_logloss: 0.437625\n",
      "[228]\ttrain's binary_logloss: 0.279592\tval's binary_logloss: 0.437633\n",
      "[229]\ttrain's binary_logloss: 0.279303\tval's binary_logloss: 0.437702\n",
      "[230]\ttrain's binary_logloss: 0.278863\tval's binary_logloss: 0.437728\n",
      "[231]\ttrain's binary_logloss: 0.278114\tval's binary_logloss: 0.437751\n",
      "[232]\ttrain's binary_logloss: 0.277583\tval's binary_logloss: 0.437956\n",
      "[233]\ttrain's binary_logloss: 0.277318\tval's binary_logloss: 0.438045\n",
      "[234]\ttrain's binary_logloss: 0.276707\tval's binary_logloss: 0.437967\n",
      "[235]\ttrain's binary_logloss: 0.2761\tval's binary_logloss: 0.438425\n",
      "[236]\ttrain's binary_logloss: 0.275484\tval's binary_logloss: 0.438495\n",
      "[237]\ttrain's binary_logloss: 0.275111\tval's binary_logloss: 0.438675\n",
      "[238]\ttrain's binary_logloss: 0.274511\tval's binary_logloss: 0.438652\n",
      "[239]\ttrain's binary_logloss: 0.273894\tval's binary_logloss: 0.438641\n",
      "[240]\ttrain's binary_logloss: 0.273434\tval's binary_logloss: 0.438684\n",
      "Early stopping, best iteration is:\n",
      "[140]\ttrain's binary_logloss: 0.337805\tval's binary_logloss: 0.435102\n",
      "tr-logloss:  0.01253347064922472\n",
      "val-logloss:  0.31119462612062593\n",
      "[9.79618677e-01 3.56919973e-01 4.69387468e-02 1.14859507e-02\n",
      " 1.05509467e-02 6.39457799e-01 5.40628075e-02 2.85946933e-03\n",
      " 9.39182398e-04 3.65430285e-01]\n"
     ]
    }
   ],
   "source": [
    "# 训练和预测\n",
    "gbdt_lr_model(data.copy(), category_fea, continuous_fea)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
